fix expert_parallel_size to not pass through to vLLM args#1181
fix expert_parallel_size to not pass through to vLLM args#1181copybara-service[bot] merged 1 commit intomainfrom
Conversation
khatwanimohit
requested review from
abheesht17,
hgao327,
jiangyangmu,
lc5211,
sizhit2,
tianshub and
wang2yn84
as code owners
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request resolves an issue where the Tunix-specific expert_parallel_size configuration was being incorrectly passed directly to the vLLM engine arguments, which does not recognize this parameter. The changes ensure that this parameter is properly consumed and translated by Tunix's internal configuration processing, preventing runtime errors and maintaining compatibility with vLLM's API. A new regression test has been added to validate this behavior. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
khatwanimohit
force-pushed
the
mohit/ep_Fix
branch
from
e82c781 to
8595728
Compare
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request correctly fixes a bug where the expert_parallel_size parameter was being improperly passed to the vLLM engine, which would cause a crash. The fix involves removing the parameter from the arguments dictionary before it's passed to vLLM. A comprehensive regression test has been added to verify the fix and prevent this issue from recurring. The changes are logical and well-implemented. I have one minor suggestion regarding a comment to better align with the repository's style guide.
khatwanimohit
force-pushed
the
mohit/ep_Fix
branch
from
82a6063 to
f96d9d5
Compare
|
Can you squash the commits please? |
khatwanimohit
force-pushed
the
mohit/ep_Fix
branch
from
f96d9d5 to
b8d95e2
Compare
3 participants
Fix (tunix/generate/vllm_sampler.py): Added args.pop("expert_parallel_size", None) at the top of _vllm_config, immediately after copying _processed_engine_kwargs. This strips the key before it can
reach LLM(**self.args) / EngineArgs(**engine_kwargs), while ep is still correctly derived via resolve_parallelism_sizes and placed into
additional_config["sharding"]["sharding_strategy"]["expert_parallelism"].Test (tests/generate/vllm_sampler_test.py): Added test_expert_parallel_size_via_engine_kwargs_not_leaked_to_vllm which passes expert_parallel_size=2 via engine_kwargs and asserts it does not appear as a top-level key in sampler.args — confirmed failing before the fix (the key was present in args).
Reference
Colab Notebook
Checklist